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ABSTRACT 



A system and process for computing a 3D reconstruction of 
a scene using multiperspective panoramas. The reconstruc- 
tion can be generated using a cylindrical sweeping approach, 
or under some conditions, traditional stereo matching algo- 
rithms. The cylindrical sweeping process involves project- 
ing each pixel of the multiperspective panoramas onto each 
of a series of cylindrical surfaces of progressively increasing 
radii. For each pixel location on each cylindrical surface, a 
fitness metric is computed for all the pixels projected 
thereon to provide an indication of how closely a prescribed 
characteristic of the projected pixels matches. Then, for each 
respective group of corresponding pixel locations of the 
cylindrical surfaces, it is determined which location has a 
fitness metric that indicates the prescribed characteristic of 
the projected pixels matches more closely than the rest. For 
each of these winning pixel locations, its panoramic coor- 
dinates are designated as the position of the portion of the 
scene depicted by the pixels projected to that location. 
Additionally, in some cases a sufficiently horizontal epipolar 
geometry exists between multiperspective panoramas such 
that traditional stereo matching algorithms can be employed 
for the reconstruction. A symmetric pair of multiperspec- 
tives panoramas produces the horizontal epipolar geometry. 
In addition, this geometry is obtained if the distance from the 
center of rotation to the viewpoints used to capture the 
images employed to construct the panorama is small in 
comparison to the distance from the center of rotation to the 
nearest scene point depicted in the images, or if an off- axis 
angle is kept small. 

61 Claims, 13 Drawing Sheets 
(2 of 13 Drawing Sheet(s) Filed in Color) 
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STEREO RECONSTRUCTION FROM parallax available in the scene. Each multiperspective pan- 

MULTIPERSPECTIVE PANORAMAS orama is essentially constructed by constraining camera 

motion to a radial path around a fixed rotation center and 

taking a single perspective image of the scene at a series of 

BACKGROUND 5 consecutive rotation angles around the center of rotation. A 

particular columnar portion of each of the single-perspective 

1. Technical Field images captured at each consecutive rotation angle is then 

The invention is related to stereo reconstruction rebinned to form the multiperspective panorama. This 

approaches, and more particularly to a system and process columnar portion of the single-perspective image can be of 

for computing a dense 3D reconstruction associated with a 1Q any width, however, it is preferred that it is one pixel column 

panoramic view of a surrounding scene using multiperspec- wide. The column may also have any available height 

tive panoramas derived from a collection of single- desired, but it is preferred the height correspond to the 

perspective images. vertical field of view of the associated single-perspective 

2 Background Art image. Note that each of these columns would have been 

* 1* ^ captured at a different viewpoint, thus the name "multiper- 

Traditional stereo reconstruction begms with two call- 15 r *\ V . . . 4 , . ; L . 

... • t , • i_ 1 t spective panorama. In addition, it is important to note that 

brated perspective images taken with pinhole cameras. To * c r . . „ , „ ' r . . , - , 

. .l. ->tn c •* the aforementioned particular column selected from each 

reconstruct the 3D position of a point in the first image, its - , . . f. A _ Al _ w . 

,. * * l . u ? j of the single-perspective images to form the multiperspec- 

corresponding point in the second image has to be round . 0 r *, 4 . f 4 . 4 , - . r . r , , 

1- r 1 ■ * • w n *• u *u tive panorama refers to the fact that each of the selected 

before applying tnangulation. Perspective cameras have the * t , , *, j ♦ *u 1 1 

. .xl * j- * * i- * - Li 1- ™ columns must have been captured at the same angle relative 

property that corresponding points he on straight lines, 20 * ^ 

which are called epipolar lines. In order to simplify the o e camera ens. 

search for correspondences, the two images can optionally Single-perspective images having the attributes discussed 

be rectified so that epipolar lines become horizontal. ab( > ve are preferably captured in one of two ways. In a first 

„ , „ ^ technique, slit cameras (or regular^ perspective cameras 

Recently, there has been a lot of work on 3D reconstruc- where 4 Qnl the center « c V olumn » fe used) are mounted on a 

tion from large collections of images Multi-basehne stereo 25 ^ ^ ^ ^ ^ ^ rf 

using several images can produce better depth maps by rQtation The cameras afe difected ^ ag tQ face endicu . 

averaging out noise and reducing ambiguities [12]. Space ^ tQ ^ ^ aQd ^ t tQ ^ ^ formed b ^ 

sweeping approaches which project multiple images onto a camefa when ^ baf ig rQtated £ach ^ ^ ^ ed 

series of imaging surf aces (usually planes), also use sigmfi- at a k of ^ t tQ ^ rotation 

cant data redundancy for better reconstruction [3, 18, 22, 9]. and each fadial positkm QQ ^ baf ^ used tQ captUfe 

Consider the problem of building^ a 3D environment _ images that will be used to construct a separate multiper- 

modei from thousands of images" captured on video. Many "spective panorama, ft is noted that a single camera could 

modeling approaches to date have concentrated on coarse ^ t> e employed and repositioned at a different radial 

reconstruction using structure from motion with a small 35 position prior to each complete rotation of the bar. As the 

number (typically hundreds) of tracked feature points. What images captured by this method are all concentric, the 

is really desired, however, are truly photorealistic multiperspective panoramas constructed are referred to as 

reconstructions, and these require dense 3D reconstruction. concentric panoramas or mosaics. 

One attempt at generating dense 3D reconstructions ^ other pre f erre d technique for capturing the desired 

involved computing a depth map for each input image [21]. ^ single-perspective images employs a "regular" perspective 

However, this method is computationally expensive. camera mounted on a rotating bar or table looking outwards. 

Granted, the expense could be lowered by sub-sampling the Here again? each image fe captured at a different angle of 

input frames (e.g., by simply dropping neighboring frames). rQtation ^ respect tQ the rotation center However, in this 

However, this risks not having enough overlapping frames case> the are nptoKd at the same radial distance 

to build good correspondences for accurate 3D reconstruc- 45 from the center of rotation> and each ^ havin g 

tl0n - the same off-axis angle from each image is used to construct 

It is noted that in the preceding paragraphs, as well as in a different one of the multiperspective panoramas. The 

the remainder of this specification, the description refers to off-axis angle is the angle formed between a line extending 

various individual publications identified by a numeric des- from the viewpoint of a column towards the lateral center- 

ignator contained within a pair of brackets. For example, 5Q une of the portion of the scene depicted by the column and 

such a reference may be identified by reciting, "reference a swing line defined by the center of rotation and the 

[1]" or simply "[1]". Multiple references will be identified viewpoint. For example, the 20* image column taken from 

by a pair of brackets containing more than one designator, ea ch image may be used to form a particular multiperspec- 

for example, [3, 18, 22, 9]. A listing of the publications tive panorama. This type of multiperspective panorama has 

corresponding to each designator can be found at the end of 55 been dubbed a swing panorama. 

the Detailed Description section. Multiperspective panoramas make it simple to compute 

SUMMARY ^ reconstructions associated with a panoramic image of the 

scene. Specifically, these reconstructions can be generated 

The present invention involves a new approach to com- using a novel cylindrical sweeping approach, or if condi- 

puting a 3D reconstruction of a scene and associated depth go tions are right, traditional stereo matching algorithms, 

maps from two or more multiperspective panoramas. These The cylindrical sweep process involves first projecting 

reconstructions can be used in a variety of ways. For each pixel of the multiperspective panoramas being used to 

example, they can be used to support a "look around and compute the depth map onto each of a series of cylindrical 

move a little" viewing scenario or to extrapolate novel views surfaces of progressively increasing radii. The radius of each 

from original panoramas and a recovered depth map. 65 0 f these cylindrical surfaces also exceeds the radius of the 

The key to this new approach is the construction and use outermost multiperspective panorama employed in the pro- 
of multiperspective panoramas that efficiently capture the cess. It is noted that the change in the radius for each 
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consecutive cylindrical surface should be made as small as employed to compute the desired reconstruction. The cir- 

possible, in light of computational limitations, so as to cumstances that ensure this approximately horizontal epi- 

maximize the precision of the resulting reconstruction. The polar geometry are as follows. First, the distance from the 

projection of the pixels of each multiperspective panorama center of rotation to the viewpoints used to capture the 

onto a particular cylindrical surface simply consists of a 5 images employed to construct the panorama should be no 

horizontal translation and a vertical scaling. Next, for each more tnan 0 J of the distance from the center of 

pixel location on each cylindrical surface, a fitness metric is rotation of each multiperspective panorama to the nearest 

computed for all the pixels projected from each multiper- point depicled m & G images. Second, in the case of a 

spective panorama onto the pixel location. This fitness swing panorama, the off-axis angle should be less than about 

metric provides an indication as to how closely a prescribed 10 15 degrees. If either of these conditions are met, current 

characteristic of the projected pixels match each other. Then, mu lti-image stereo matching algorithms can be applied 

for each respective group of corresponding pixel locations of without modification to compute the depth maps (or if only 

the cylindrical surfaces, it is determined which particular two mu itip ersp ective panoramas are available, the less accu- 

location of the group has a computed fitness metric that rate two -image based stereo matching algorithms can be 

indicates the prescribed characteristic of the projected pixels 15 used ) It ^ noted mat m me case of a swing panoramaj { t ma y 

matches more closely than the rest. This will be referred to be nec essary to compensate for a global vertical scaling 

as the winning pixel location. Specifically, the winning pixel differences caused by a non-zero off-axis angle before 

location can be determined using any appropriate applying the aforementioned traditional algorithms, 

correlation-based or global optimization technique If m addition to the just described benefits, other advantages 

correlation-based i methods are employed, U is preferred that 20 t ^ become from * 

this entail spatially aggregating the computed fitness metric detaile d P description which follo ws hereinafter when taken in 

of each pixel location on each respective cylindrical surface ■ *• vu *u j • c u- u 

. r . . c . r * • , , conjunction with the drawing figures which accompany it. 

using the computed metrics of its neighboring pixel loca- J & & r J 

tions. DESCRIPTION OF THE DRAWINGS 

The theory behind this cylindrical sweep approach is that 2 s m „, , . . , , 

the pixels in each multiperspective panorama, which depict ^ ^ e of this patent contains at least one drawing 

the same portion of the scene, will converge to the same executed in color Copies of this patent with color drawing 

location on one of the candidate cylindrical surfaces at the < s ) ^ be P rovid , ed bv the U f P atent and Trademark Office 

depth associated with that portion of the scene. For each U P° D re <l uest ««* P avmeDt of lhe necessary fee. 

winning pixel location, its panoramic coordinates are iden- 30 T°e specific features, aspects, and advantages of the 

tified and these coordinates are designated to be the position present invention will become better understood with regard 

of the portion of the scene depicted by the pixels projected . to the following description, appended claims, and accom- 

from the multiperspective panoramas to that location. panying drawings where: 

Finally, a depth map can be generated via conventional FIG. 1 is a diagram depicting a general purpose comput- 
methods from the panoramic coordinates. It is also noted 35 ing device constituting an exemplary system for implement- 
that if the cylinders are processed in front to back order, ing the present invention. 

occlusion relationships can also be determined via conven- FIG. 2 is a flow chart diagramming an overall process for 

tional methods. computing a dense 3D reconstruction associated with a 

It was mentioned earlier that if conditions are right, panoramic view of a surrounding scene using multiperspec- 

traditional stereo matching algorithms can also be employed 40 tive panoramas according to the present invention, 

to compute the desired depth map. One example of the right FIG. 3 is a diagram showing a side view of a simplified 

conditions involves the use a pair of symmetric multiper- f or capturing slit images of a real 3D scene using 

spective panoramas. A symmetric pair of multiperspective cameras. 

panoramas in the case of swing panoramas is one where the pjQ. 4 is a diagram showing concentric circles on the 

off-axis angles are symmetric with respect to the swmg line. 45 circle plane and concentric panoramas associated with each 

In the case of concentric panoramas, a symmetric pair is one circle. 

where each panorama was generated from images taken by 1-1^0 e/ \ j e/u\ • j • 

F . , c . . « . . . J FIGS. 5(a) and 5(b) are images depicting concentric 

cameras respectively facing in opposite direction at the same \ / . . *■ v • » 7 * * a>% . 1 

,. * r . r a . • r panoramas derived from im ages taken at two different radial 

radius from the center of rotation. A symmetric pair of f ™„ v . r , . . , . 

tx . , . , ' . . T . t . locations. FIG. 5(c) is an image depicting a depth map 

multiperspective panoramas has the characteristic that the 50 t j r *u * * cm no e , \ a 

. , r t . . r . ■ A l computed from the concentric panoramas ot FIGS. 5(a) and 

epipolar geometry consists of horizontal lines. As such, any -A . j •*• 

. * J . . , 1 . 5(6), among others. FIG. 5(a) is an image depicting a 

traditional stereo algorithm requiring this horizontal epipo- v . 0 tl _ . , c 0 , A. 0 c 

. , & . . « . , « \ t_ concentric panorama re -synthesized from the depth map of 

lar geometry (e.g., a hierarchical warp algorithm) can be ~ir> */ \ 01^0 e/ \ *u u \ -~ a ■ r 

1 j. '.i.j-j * *• jj.u FIG- 5(c). FIGS. 5(e) through 5(e) are images depicting 

employed to compute the desired reconstruction and depth . w I r*r*a */ \ j *fL\ m>- c/iX • 

v J F * close-up portions of FIGS. 5(a) and 5(b), FIG. 5(h) is an 

ma P s " , , . 55 image depicting a close-up portion of FIG. 5(d). 

However, it is desirable that more than two multiperspec- , ~ , , ■ . . r 

, j HG.oisa side elevational view or another camera setup 

tive panoramas be used to compute the depth map in order ,. . , . . „ ... . r 

/I . . j u * j u ., accordmg to the present mvention for providmg swing 

to obtain a more accurate and robust correspondence, while u * *i * .j ♦ 

.„ . . . . *■ 11 • . 1 panoramas wherem a smgle camera is mounted on a rotating 

still employing the less computationally intense stereo algo- T m t ti 1 

rithms. Such multi-image stereo matching algorithms do 60 tat)le t0 capturc a real scene * 

exist, but require a horizontal epipolar geometry between the FIG - 7 shows a t0 P view of me camera and rotatin S table 

input images. Fortunately, it has been discovered that under °^ 6. 

certain circumstances the epipolar geometry between the FIG. 8 is a diagram illustrating a sequence of images 

multiperspective panoramas produced in accordance with captured by the camera of FIG. 6. 

the methods of the present invention do sufficiently approxi- 65 FIG. 9 is a diagram showing the path of a camera on a 

mate the required horizontal lines that the aforementioned plane and swing panoramas derived from images captured 

traditional multi-image stereo matching algorithms can be on the path. 
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FIGS. 10(a) and 10(6) are images depicting swing pan- input/output system 26 (BIOS), containing the basic routine 

oramas derived from image columns associated with two that helps to transfer information between elements within 

different off-axis angles. FIG. 10(c) is an image depicting a the personal computer 20, such as during start-up, is stored 

depth map computed from the swing panoramas of FIGS. ^ ROM 24. The personal computer 20 further includes a 

10(a) and 10(6), among others. FIG. 10(d) is an image 5 hard disk drive 27 for reading from and writing to a hard 

depicting a novel panorama extrapolated from the depth map disk * not shown, a magnetic disk drive 28 for reading from 

of FIG. 10(c). FIGS. 10(e) through 10(g) are images depict- or writing to a removable magnetic disk 29, and an optical 

ing close-up portions of FIGS. 10(a) and 10(b). FIG. 10(h) d isk drive 30 for reading from or writing to a removable 

is an image depicting a close-up portion of FIG. 10(d). optical disk 31 such as a CD ROM or other optical media. 

FIG. 11 is a diagram showing the geometric and trigone- io The hard disk drive 27, magnetic disk drive 28 and optical 

metric relationships for a multiperspective panorama. d * k 30 connected to the system bus 23 by a hard 

t-t^^- a l^j- • d ^sk drive interface 32, a magnetic disk drive interface 33, 

FIG. 12 is a flow chart chagramrmng a cyhndncal sweep- and an ^ interfac f 1 34> respectively . ^ 

me process according to the present invention for accom- , . . , , , , , . . 

,f *, . & . r , , - it _ » c and their associated computer-readable media provide non- 

phshmc the reconstruction module of the overall process of , ... , c * . j ui • * j* 

- & 15 volatile storage of computer readable instructions, data 

' ' structures, program modules and other data for the personal 

FIGS. 13 and 14 are graphs plotting horizontal parallax computer 2 0. Although the exemplary environment 

for varying values of $ and R, respectively. described herein employs a hard disk, a removable magnetic 

FIGS. 15 and 16 are graphs plotting vertical parallax for disk 29 and a removable optical disk 31, it should be 

varying values of $ and R, respectively. 2Q appreciated by those skilled in the art that other types of 

FIGS. 17A and 17B are flow charts diagramming an computer readable media which can store data that is 

expanded overall process for computing a dense 3D recon- accessible by a computer, such as magnetic cassettes, flash 

struction associated with a panoramic view of a surrounding memory cards, digital video disks, Bernoulli cartridges, 

scene using multiperspective panoramas according to the random access memories (RAMs), read only memories 

present invention. 25 (ROMs), and the like, may also be used in the exemplary 

DETAILED DESCRIPTION OF THE operating environment. 

PREFERRED EMBODIMENTS .. A n^ber of program modules may be stored °n the hard 

disk, magnetic disk 29, optical disk 31, ROM 24 or RAM 25, 

In the following description of the preferred embodiments including an operating system 35, one or more application 

of the present invention, reference is made to the accorapa- 30 programs 36, other program modules 37, and program data 

nying drawings which form a part hereof, and in which is 38. A user may enter commands and information into the 

shown by way of illustration specific embodiments in which personal computer 20 through input devices such as a 

the invention may be practiced. It is understood that other keyboard 40 and pointing device 42. Of particular signifi- 

embodiments may be utilized and structural changes may be cance to the present invention, a camera 55 (such as a 

made without departing from the scope of the present 35 digital/electronic still or video camera, or film/photographic 

invention. scanner) capable of capturing a sequence of images 56 can 

FIG. 1 and the following discussion are intended to also be included as an input device to the personal computer 

provide a brief, general description of a suitable computing 20. The images 56 are input into the computer 20 via an 

environment in which the invention may be implemented. appropriate camera interface 57. This interface 57 is con- 

Although not required, the invention will be described in the 40 nected to the system bus 23, thereby allowing the images to 

general context of computer-executable instructions, such as be routed to and stored in the RAM 25, or one of the other 

program modules, being executed by a personal computer. data storage devices associated with the computer 20. 

Generally, program modules include routines, programs, However, it is noted that image data can be input into the 

objects, components, data structures, etc. that perform par- computer 20 from any of the aforementioned compute r- 

ticular tasks or implement particular abstract data types. 45 readable media as well, without requiring the use of the 

Moreover, those skilled in the art will appreciate that the camera 55. Other input devices (not shown) may include a 

invention may be practiced with other computer system microphone, joystick, game pad, satellite dish, scanner, or 

configurations, including hand-held devices, multiprocessor the like. These and other input devices are often connected 

systems, microprocessor-based or programmable consumer to the processing unit 21 through a serial port interface 46 

electronics, network PCs, minicomputers, mainframe 50 that is coupled to the system bus, but may be connected by 

computers, and the like. The invention may also be practiced other interfaces, such as a parallel port, game port or a 

in distributed computing environments where tasks are universal serial bus (USB). A monitor 47 or other type of 

performed by remote processing devices that are linked display device is also connected to the system bus 23 via an 

through a communications network. In a distributed com- interface, such as a video adapter 48. In addition to the 

puting environment, program modules may be located in 55 monitor, personal computers typically include other periph- 

both local and remote memory storage devices. eral output devices (not shown), such as speakers and 

With reference to FIG. 1, an exemplary system for imple- printers, 
menting the invention includes a general purpose computing The personal computer 20 may operate in a networked 
device in the form of a conventional personal computer 20, environment using logical connections to one or more 
including a processing unit 21, a system memory 22, and a 60 remote computers, such as a remote computer 49. The 
system bus 23 that couples various system components remote computer 49 may be another personal computer, a 
including the system memory to the processing unit 21. The server, a router, a network PC, a peer device or other 
system bus 23 may be any of several types of bus structures common network node, and typically includes many or all of 
including a memory bus or memory controller, a peripheral the elements described above relative to the personal corn- 
bus, and a local bus using any of a variety of bus architec- 65 puter 20, although only a memory storage device 50 has 
tures. The system memory includes read only memory been illustrated in FIG. 1 The logical connections depicted 
(ROM) 24 and random access memory (RAM) 25. A basic in FIG. 1 include a local area network (LAN) 51 and a wide 
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area network (WAN) 52. Such networking environments are panorama, however, each of these columnar images repre- 

commonplace in offices, enterprise-wide computer sents a portion of the scene captured from a different 

networks, intranets and the Internet. perspective. Thus, multiperspective panoramas differ from 

When used in a LAN networking environment, the per- conventional panoramas in that parallax effects are captured 
sonal computer 20 is connected to the local network 51 s *ie «° the varying perspective of the columnar images 
through a network interface or adapter 53. When used in a Specifically, each of the columnar images is derived from 
ii /a \y i * - 4 i * ->a a different one of the single-perspective images and each 
WAN networiang environment, the personal computer 20 sin&le -perspective ; £ ca P mred at differ | nt rotational 
typically includes a modem 54 or other means for establish- ^ { ^ cQm £ Qa centef Qf rolation ^ ^ 
ing communications over the wide area network 52, such as accomplished by constraining camera motion to a radial path 
the Internet. The modem 54, which may be internal or ™ around a fixed rotation center and taking a single perspective 
external, is connected to the system bus 23 via the serial port j mage 0 f the scene at a series of consecutive rotation angles 
interface 46. In a networked environment, program modules around the center of rotation. A particular columnar portion 
depicted relative to the personal computer 20, or portions 0 f each of the single-perspective images captured at each 
thereof, may be stored in the remote memory storage device. consecutive rotation angle is then extracted (or captured as 
It will be appreciated that the network connections shown 15 a slit image using line scan sensors such as a linear push- 
are exemplary and other means of establishing a communi- broom camera [5]) and rebinned to form the multiperspec- 
cations link between the computers may be used. tive panorama. The idea of resampling and rebinning has 

The exemplary operating environment having now been recently become popular in image-based rendering. For 
discussed, the remaining part of this description section will exam P le ' * c Lumigraph and Lightfield resample all cap- 
be devoted to a description of the program modules embody- 20 mrcd n g r ^ m them mt0 a 4D two-plane parameter- 
ing the invention. Generally, the system and process accord- ™ [ ™ t 4 ' 10 1 ^ re f°° f ° r Winning input images is to 

• , ,u * * • _ i „ „ * r „ . . find a representation that facilitates the desired application. 

me to the present invention involves a new approach to _ . y . ...... * r , 

, ™ 4 _ '^/j - tU ~ For the Lumigraph, this is the re -synthesis of novel views. 

computing a dense 3D reconstruction associated with a _ _ ..... . ... ... 

panoramic view of a surrounding scene using multiperspec- However, this idea has never been applied to model-based 

live panoramas derived from a collection of single- 25 'Potions such as the present 3D reconstruction of a scene 

perspective images. Only a small number of multiperspec- and ,he ^PuUtion of depth maps form the reconstruction 

r ° „ , . „u*„:„ „ f L ncn nn A in It is noted that the aforementioned columnar portion of 

tive panoramas are needed to obtain a dense and accurate 3D . . . . e ... . 

reconstruction, since the panoramas sample uniformly in !* e smgle-perspectiye image car. be of any width^ however, 

three dimensions: rotation angle, inverse radial distance, and A 18 P r , efe " ed ^ 11 15 one P^ 1 5*"? n wide. TTie column 

vertical elevation. Unlike a collection of single-perspective 30 *° ? ave any availab ^ W.?""? 4 .^^ 1 

panoramas taken from different locations, there are no k Pf ,he correspond to the full vertical field of view 

preferred directions or areas where the matching fails of * e single-perspective images. 

because the disparity vanishes. Using multiperspective pan- .11* multiperspective panoramas generated in accordance 

, ., .. - t , . i_ -i with the present invention are somewhat related to known 

oramas also avoids the limited overlap between the original " F r , " . ^ 

... .i mr •„ „„,.„,, J~! u; 35 configuration such as multiperspective panoramas for eel 

input images that causes problems in conventional multi- . *>. . . . *\ r . . r . r 

u v 7 Tn, i* n ™„u animation [24], multiple-center-of-images [16], manifold 

baseline stereo. The multiperspective panorama approach . _ \. J ' . *\ . . J . t . 4 

further differs from stereo matching of panoramic images mosa ' cs P 4 1 and c^cular projections [13 and related to 

taken from different locations, where the epipolar con- ma 8« s ° b amed b * P"* broom cameras [5] and panning 

straints are sine curves, in that for multiperspective "on-frontal imaging cameras [8]. However, unlike these 

panoramas, the epipolar geometry, to first order" consists of 40 M>P™ches, the present mulnperspective panoramas are 

horizontal lines. TCerefore, any traditional stereo algorithm P roduced bv , °°!» tramm g me camera mo ! ,ons al ° n g. radia 

. . , ..o paths around a fixed rotation center. It is this constraint that 

can be applied without modification. ... c • . * 

rr enables the use of a simple surface sweep stereo correspon- 

In general, this approach to computing a dense 3D recon- deflce ^ tQ compute a 3D reconstruction for the scene, 

struction using multiperspective panoramas is accomplished 45 and under certaifl COIlditions results m a horizont al epipolar 

via the following process actions, as shown in the high-level geom e tr y that allows the use of traditional stereo reconstruc- 

flow diagram of FIG. 2: tion algorithms to compute the reconstruction. 

a) constructing two or more multiperspective panoramas While single-perspective images having the attributes 
from the collection of single-perspective images of the discussed above can be capture in any appropriate manner, 
scene (process action 200); and, 50 it is preferred that the methods disclosed in a co-pending 

b) computing a 3D reconstruction of the scene from the patent application be employed (also see reference [20]). 
multiperspective panoramas (process action 202). The co-pending application is entitled RENDERING WITH 

A description of what a multiperspective panorama is and CONCENTRIC MOSAICS and assigned to the common 

details as to their construction will be provided in the first of assignee hereto. The application was filed on May 11, 1999 

the following sections (Section 1.0). Then, in Section 2.0, 55 and assigned Ser. No. 09/309,753. The subject matter of this 

the ways in which these multiperspective panoramas can be co-pending application is hereby incorporated by reference, 

used to compute a reconstruction of the scene will be The co-pending application primarily describes two camera 

described. Finally, experimental results associated with rigs that can be used to capture the single-perspective 

tested embodiments of the system and process will be images needed to produce the multiperspective panoramas 

provided in Section 3.0, and alternate techniques are 60 associated with the present invention. The first rig uses 

described in Section 4.0. several "slit cameras" (or "regular" perspective cameras 

1.0 The Multiperspective Panorama where only a single pixel column is kept) mounted on a 

A multiperspective panorama is essentially a 360 degree rotating bar such that their optical axis is tangent to the circle 

panoramic view of a surrounding scene made up of a traced out by the bar. The other camera rig uses a single 

plurality of side-by-side columnar images depicting con- 65 perspective camera mounted on a rotating bar looking 

secutive portions of the scene derived from a collection of outwards. The particulars of each of these configuration will 

single-perspective images. Unlike a typical mosaic now be described in the sub-sections below. 



04/12/2004, EAST version: 1.4.1 



US 6,639,596 Bl 

9 10 

1.1 Concentric Panoramas or Mosaics plane, in a direction tangent to the circle. These slit images 
As mentioned above, one way of accomplishing the 404 captured by the camera rotation along each circle 400 
constrained camera motion needed to capture the required are combined in an side-by-side manner to form respective 
single-perspective images is to employ the first of the concentric panoramas, such as CM k and CM„ shown in FIG. 
aforementioned camera rigs shown in FIG. 3. This first rig 5 4. These concentric panoramas are illustrated in an 
preferably entails mounting several cameras 300 (C 0 , . . . "unwrapped" form by the rectangles 402. 
C k . . . ,C„) on a rotating horizontal beam 302. While, only To this point, the process of creating concentric panora- 
three cameras 300 are shown in FIG. 3, there would actually mas of a surrounding 3D scene has involved capturing 
be many more positioned along the rotating beam. In the images of a real scene using cameras. However, the present 
depicted case, the beam 302 is raised up from "ground level" 10 invention is not limited to generating the concentric mosaics 
by a support 304. This expands the vertical field of view. The from a real 3D scene. The concentric mosaics can also be 
interface between the support 304 and the beam 302 is generated synthetically using any appropriate graphics pro- 
configured to allow the beam to be rotated a full 360 degrees. gram capable of producing a 3D scene. Essentially, these 
Thus, this interface defines a center of rotation 306. The programs can be employed to generate slit images of the 
plane defined by the circular region swept out by rotating the is desired dimensions, which depict a portion of a surrounding 
beam 360 degrees is a circle plane (which in the depicted synthetic 3D scene as would be captured by a camera 
case is a horizontally oriented plane), and the circles traced rotating at radius R from the center of rotation. Accordingly, 
out by the focal point of each camera on the beam when the in the description of computing a 3D reconstruction from the 
beam is rotated constitute concentric circles on the circle concentric panoramas that is to follow, it should be under- 
plane. The depicted multiple camera configuration is pre- 20 stood that the captured slit images can refer to either real 
ferred as several of the concentric mosaics can be created images or synthetic images. 

simultaneously. However, it would also be feasible to use Examples of synthetic concentric panoramas generated as 
just one camera and move its radial location on the beam described above can be seen in FIGS. 5(a) and (b), each of 
prior to each rotation of the beam. which represents what would be captured with cameras 
The cameras 300 may be so-called "slit" or "vertical line" 25 having normalized radii of 1.0 and 0.7, respectively, 
cameras in that only the center line of each image of the 3D Additionally, FIGS. 5(e) through (g) depict close-ups of 
scene is captured. These images will hereinafter be referred portions of the synthetically generated concentric panora- 
to synonymously as "slit" images, image columns, or image mas showing the horizontal parallax captured in the pan- 
lines. Preferably, the prescribed width of the slit images is as oramas. 
narrow as possible — ideally the width of a single pixel. 30 1.2 Swing Panoramas 

Alternatively, standard perspective cameras can be The other preferred way of accomplishing the constrained 
employed. Such cameras will produce an image of the scene camera motion needed to capture the required multi- 
having some horizontal field of view. However, only the perspective images is to employ the alternate camera rig 
center portion of this horizontal field will be used, and shown in FIG. 6. This camera configuration essentially 
preferably only the single, center pixel column is used. This 35 involves swinging a "regular" perspective camera mounted 
center portion or column is extracted from the perspective on a rotating bar or table looking outwards. In this case, 
image via conventional methods. different image columns of the resulting single-perspective 

A multiperspective panorama is constructed by collecting images are used to construct different multiperspective pan- 
slit images at a series of rotation angles as the camera is oramas. And more particularly, the image column occupying 
rotated a full 360 degrees. The images are captured with a 40 the same relative horizontal location in each single perspec- 
frequency, which given the prescribed width of the images tive image is used to construct a multiperspective panorama, 
and the speed of rotation, will produce images that when These panoramas are called swing panoramas, 
placed side-by-side in the order they were taken, form a Specifically, referring to FIG. 6, the camera set-up 600 
complete multiperspective panoramic view of the surround- uses a single camera 602 that is moved (i.e., swung) along 
ing scene. Ideally, the camera motion is made continuous 45 a circular path in a plane. In particular, the camera is 
and at a uniform speed for the full 360 degrees of rotation. mounted to a bar or table 604 that rotates the camera 602 at 
This will ensure the images will line up perfectly with no a radius R about a center point 606 (where R is measured 
overlaps or gaps. The multiperspective panoramas created from the center point to the camera's focal point). As 
with the foregoing camera rig are called concentric panora- indicated by arrow 610, the table is rotated in a clockwise 
mas or mosaics because each is derived from a camera 50 direction at constant speed, but may also be rotated in the 
rotating along one of several concentric circular paths. counterclockwise direction if desired. 

The process of creating a series of consecutive slit images Referring now to the top view of the swing panorama 

that collectively depict the surrounding 3D scene is per- camera rig 700 shown in FIG. 7, the camera 702 moves 

formed for each of the concentric circles 400 on the circle along a circular path 704 defined by the focal point of the 

plane to form a collection of concentric panoramas or 55 camera and has its viewing direction positioned perpendicu- 

mosaics 402, as shown in FIG. 4. In the illustration of the lar to the circular path (as opposed to tangent as in the case 

process in FIG. 4, the left hand side shows a series of of a concentric panorama camera rig). As indicated at 706, 

concentric circles including the center of rotation C 0 (i.e., a the camera 702 has a constant horizontal field of view. Light 

circle with a radius of zero), one of the intermediate circles rays enter the camera 702 at different angles or ray directions 

C& and finally an outermost circle C M . Slit images are 60 across the entire horizontal field. For the purposes of 

captured by moving a camera along each of the concentric illustration, a light ray farthest to the right of the camera's 

circles and capturing the view of the 3D scene in directions horizontal field of view 706 is designated as angle 

tangent to one of the concentric circles in the circle plane. ((>=-MAX. Similarly, a light ray entering on the farthest left 

The geometry of this procedure is best seen in the right-hand side of the horizontal field of view 706 has a ray direction 

side of FIG. 4. As can be seen, each slit image L (depicted 65 designated as <|>«MAX, wherein the values of <(>«MAX and 

as lines 404) is captured at a viewpoint v (identified by the MAX are a function of the camera used. Centrally 

short radial lines 406) on a concentric circle 400 in the circle located in the horizontal field of view 706 is the camera 
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viewing direction designated as 4>-0. As the camera 702 circumstances. These are needed in turn to describe how a 

rotates along the circular path, it position can be defined by 3D reconstruction can be generated from a set of multiper- 

an angle 8. spective panoramas. 

Referring to FIG. 8, the sequence of images 800 captured Referring to FIG. 11, a top-down view shows the center 
by the aforementioned camera are numbered such that image 5 of rotation C around which the camera rotates along a radius 

0 is the first image captured at a predetermined starting of size R - T*« camera fe located at v > and me selected 
point. For example, image 0 may be the image captured column of P ixels is lookin g at a P° int p > which fc at a 
when the camera 702 crosses the x-axis of FIG. 7. The £stonoe from the center C, and at an in-plane distance d 
images are characterized for the purposes of the present * om camera V. The current ang e of rotation is denoted 
. * . u • c j c i V. c _* ii i- a by 0, and vanes between 0 and 360 . In a panoramic image, 
mvennonasberngformedofaplurahtyofverUcaUy digned u ? ^ , he ^ Q ^ A axis. A typical rebinned panoramic 
d^ete image columns such as ™agecohimn 802, that . have mo ea^^ one taken from the 
have a height corresponding to the vertical field of view of . • c r • • * • _i 

. t_ - i i_ • 1 -rr . j- same column m successive frames of a given input video, 

the camera. Each image column has a different ray direction ™ , ,v . . . . f . , . 

ix . . j ri i a. _ a . • The otner (vertical) axis in the panoramic image is indexed 

or angle <b associated with it. For example the leftmost image i *• u / . u * *u 

i o,vi • • __ « • . 1 1 t _f ~ by the elevation or row number y (not shown in the 

column 802 in image 0 corresponds to <j>~MAX, whereas the 15 J . fi . J v 

rightmost column 804 corresponds to (b— MAX. For any °?n. 0W ? ®c i a u ^- i 

, , , , r. r • i a* The plane of pixels captured by a particular panoramic 

particular value of <b there are a plurality of pixels extendmg *1 r . , r ' . r .. . ; u . 

1 , n T, r . f . .if • ___ r *i_ imager forms an angle <b with the swing line in the normal 
for the vertical field of view to define the height of the . . _• */ -__-* nn, w i i 

T , ,. . « . j . r • i direction connectmg V and C. When multiple columns are 

image. In addition, a predetermined number of image col- , . . c • _■ . jr ^/.u « • 

& . i .jt * . . - . ? . . selected from a camera pointing outward from C (the swing 
umns make up the width of the image. The number of pixels 20 „ ~ r . f t . tX , x/r f 

jc - __. if • • t rft. i c i j * ■ panorama configuration), it follows that <b-tan ((x-xJ/f), 

defining the height is a function of the vertical field of view r , . t , % " . . . \ . vv 

r . & & j .i-- 1 _ • where x is the column number in the input image, x,, is the 

of the camera, and the number of image columns making up 4 , , c . c i i lv • . T 

lL . . . * r l -i center column, and f is the focal length m pixels. In the 

the width of the image is a function of how may pixel . c , * 0 T r A . 

& . . . j concentric panorama configuration, 6=90 . In order to gen- 

columns constitute an image column. As mentioned iL r ^ t , , , . . t * , ■ c 

. , A . r i ... 4 eralize the geometry, the angle between the optical axis of 

previously, it is preferred that each image column constitute 25 4 . j .u i u * u- a - - * a a i 

^ . 7 ' f , . . 6 the camera and the column being rebinned is introduced. In 

a column one pixel wide. . , x i_ • 

n *n . . i_ ii_ j . . j . swing panoramas, U>=<b, whereas in concentric panoramas, 

FIG. 9 illustrates how the captured images may be used to , *. r i ■ • j- r.u 

. t, f . nftrt & « .1. ■ i because while the tangential viewing direction of the 

form a swing panorama. The circle 900 represents the circle ■ .t_ . • • u x n^o iL 

tl &r . . . \ , ,v camera m the concentric panorama case yields <p«90 , the 

that the camera rotates on. At each viewpoint (e.g., v, and v ; ) , , 4 iL 4 . . r . ^ ^ ' ... 

nM nft>l . iL . . t iL v & »t 7/ angle between the optical axis of the camera and each sht 

902. 904 along the circle 900 that the camera captures an 30 . b , , . 

' - . & , . . . . image captured is 90 . 

image of the scene, the image columns making up the _° r . . . . . /Q x 

& j. i_-j.vcji.-_i j / To summarize, each panoramic image is indexed by (8,y), 

captured image can be identified by indexed rays (e.g., r^Tfr, ... .... r . °. . , , /n J \ ,( 

j \ ft Ao nm mi *u - i i u % u- u and its acquisition geometry is parameterized by (R, ((), t|j). 

and rj 908, 910, 912 on the circle plane, each of which T . & • . -. L ± , ' - 

, . , , - 5 A . In the most common acquisition setups, either <p**iiJ_«tan 

corresponds to a particular value of (p. A swing panorama, n x , „ ^ . i / x 

. l j * . j ow • r j ,_ i_ * * • i(kw), k— K ... K, where kw«(x-xJ/f (swing panoramas), 
such as the depicted SM t , is formed by combining in a 35 v , \ r . 0 , n n i n i n / . • \ 

. , , r . r , J , . & or d>=± 90 ,il)«0,R fr »kR 1 ,k-0 ... K (concentric panoramas), 

side-by-side manner the image columns 906 having the ^ 4 Anal sis of Parallax 

same angle <b, associated with ray r_- taken from the images *_ j * m £ .#■ 

^ Y ' , . , . rt r In order to compute a 3D reconstruction from a multi- 

captured at each sequential rotation angle 0. Other swmg . •* • « a c *u u * * 1 

r 1 7 * . • j-_r .1 r. perspective panorama, it is necessary to define the horizontal 

panoramas may be formed by using different values of A, j 1 n a • »u u- j • _ 

r , ._, .._.«». u- i_ - r ii_ i_- ■ .J and vertical parallax captured in these rebmned images. In 
such as the depicted SM fr which is formed by combining the 40 ... 4 . r 1 r .t_ . * .1 j 1 

1 * . j -.i_ .i_ r i_ this section, the formulas for the horizontal and vertical 

image columns 914 associated with the ray r„ from each - . it . - i-j i rj- 

& , . , ! 1 n t 1 parallax of a point located on a cylindrical surface of radius 

image captured at each sequential angle 6. In general, a |^ derived 

series of images such as a video sequence of F frames of size TT . .. . * 1 c * c .* t 

«r tt . . . . . . / . 7 w -.__ * Usmg the basic law of sines for tnangles, 

WxH can be rebmned mto (up to) W panoramas with size 0 

FxH. It is noted that, here again, the panoramas are illus- 45 r r d (l) 

trated in an "unwrapped" form by the rectangles 916 and . ljL m = . = ^-z, or 

918, respectively. -< 180 "« ^ 

FIGS. 10(a) and 10(_?) show examples of swing panora- _ ■ -i( R ■ \ ^ 

mas that were generated using a camera set-up such as 0-<p-^n [-sin^j. 

described above. The panorama shown in FIG. 10(a) was 50 

formed using the leftmost image column from each single- mref (he horizontal Uax ^ Q e for a point at 

perspective . mage, whereas the panorama of FIG. 10(6) was rsecoiatWQ p ^ noralnic £> gts \ ^ , 2 

formed usuig the center image column of the single- of a ^ a _ d ^ te _^ d< ^ endin 2 on _ , f 

perspective .mages. While the dep.cted examples were gen- each oramic ^ (circularly f by ^ fiist 

era ted from actual images of a scene, it is noted that swmg 55 - # r , , t & . v ^/ / w 

, , & . j ,» . . |, . & factor drops out, leaving 

panoramas can also be generated synthetically usmg any r & 

appropriate graphics program capable of producing a 3D 

scene, just as in the case of a concentric panorama. = sin _l (— siu^i) - sin" l (— sin^ 2 )- 

1.3 The Imaging Geometry of Multiperspective Panoramas r r 

The imaging geometry and trigonometric relationships for 60 
a single multiperspective panorama created in accordance The vertical parallax can be derived using the following 

with the present invention can be generalized as shown in observation. Recall that according to the laws of perspective 

FIG. 11. The generalized geometry and trigonometric rela- projection, the appearance (size) of an object is inversely 

tionships will be used in subsequent sections to define the proportional to its distance along the optical axis, e.g., 
horizontal and vertical parallax captured in each panorama 65 u«fx/z, v=fy/z. For pixels at a constant distance r from C, 

and to show that an essentially horizontal epipolar geometry and therefore a constant distance from V along the optical 

exist between multiperspective panoramas under certain axis, the vertical scaling can be computed directly from this 



04/12/2004, EAST Version: 1.4.1 



US 6,639,596 Bl 



13 



distance z-d cosij). Here, d is the in-plane distance between 
V and P, and d cos i|j is the distance along the optical axis 
(as mentioned before, typically i^=(j) or i|>«0). 

This scale factor can be written as s=f/z=f7(d cos 
Using the law of sines (1) again, the change of scale between 
two panoramic images can be computed as 



Expanding sin 6, where 6 is given by (2), it follows that 

sinfl = siu^cos^sifl" 1 ^ sio^jj - cos^ * sintf 



= simp 



- cos^ 



10 



15 



Thus, the scale change can be re-written as 



(4) 



25 



The first factor in this equation depends on r, and goes to 1 
as r-»oo. The second factor is a global scale that compensates 
for the off-axis angle of a given column. This is the same 
correction that is applied to rectilinear (perspective) images 
when converting them into cylindrical coordinates [23]. The 
vertical parallax for any pixel in row y x of image I a can be 
computed directly from 

Ay 2 :l-y2-yi-(52:l-l)>l- (5) 

2.0 Computing 3D Reconstruction from Multiperspective 
Panoramas 

Multiperspective panoramas make it simple to compute a 
3D reconstruction and depth map associated with a pan- 
oramic image of the scene. For example, the reconstruction 
and depth map can be generated using a unique cylindrical 
sweeping approach, or if conditions are right, traditional 
stereo matching algorithms. 

2.1 Cylindrical Sweep Approach to 3D Reconstruction 

A multi-image cylindrical sweep approach can be 
employed to compute a 3D reconstruction of the scene from 
the multiperspective panoramas. This approach is an expan- 
sion of known stereo algorithms, particularly the concept of 
plane sweep stereo. 

Plane-sweep and space coloring/carving stereo algorithms 
have recently become popular, because they support true 
multi- image matching [3], enable reasoning about occlusion 
relationships [18, 22, 9], and are more efficient than tradi- 
tional correlation-based formulations [6]. Traditional stereo 
matching algorithms pick a window around each pixel in a 
reference image and then find corresponding windows in 
other images at every candidate disparity (searching along 
an epipolar line). Plane sweep algorithms consider each 
candidate disparity as defining a plane in space, and pro- 
jecting all images to be matched onto that plane, using a 
planar perspective transform (homography) [3, 22, 1]. A 
per-pixel fitness metric is then computed which may be 
aggregated spatially. After all the cost functions have been 
computed, a winning disparity can be chosen. 

As stated above, the cylinder sweep approach according 
to the present invention preferably expands upon the plane 



35 



40 



45 



50 



55 



60 



65 



14 



sweep algorithms to allow a 3D reconstruction of a scene to 
be computed directly from a set of two or more multiper- 
spective panoramas. In general, this is accomplished by 
projecting multiperspective panoramas onto cylinders of 
varying radii r. The transformations that map panoramas 
onto these cylinders, and hence onto each other, are particu- 
larly simple. The re-projection of each panoramic image 
onto a cylindrical surface of radius r merely consists of a 
horizontal translation and a vertical scaling. The proof 
follows direcdy from equations (3) and (4) for horizontal 
and vertical parallax. A less formal but more intuitive proof 
is to just observe that the image of any column of a cylinder 
at a fixed radius r seen by a concentric pusbbroom camera 
is just a scaled version of the pixels lying on that cylinder. 
Since the multiperspective panorama representation has no 
preferred direction, the shift between the panoramic image 
and the cylinder must be the same for all pixels. 

The example cylindrical sweeping approach specifically 
involves projecting the pixels of each multiperspective pan- 
orama onto a series of concentric cylindrical surfaces each 
of which has a greater radius than the previous one, and each 
of which has a radius that exceeds all the multiperspective 
panoramas that are being used to compute the desired 
reconstruction of the scene. It is noted that the change in the 
radius for each consecutive cylindrical surface should be 
made as small as possible, in light of computational 
limitations, so as to maximize the precision of the resulting 
depth map. As indicated above, this projection of the pixels 
of each multiperspective panorama onto a particular cylin- 
drical surface simply consists of a horizontal translation and 
a vertical scaling using equations (3) and (5), respectively. 
- Referring to FIG. 12, one preferred implementation of the 
cylindrical sweep approach is presented. As can be seen, 
each multiperspective panorama that is to be used to com- 
pute the reconstruction is projected onto each cylindrical 
surface (process action 1200). For each of these cylindrical 
surfaces, an appropriate fitness metric is computed for all the 
pixels projected from each of the multiperspective panorama 
onto the same pixel location of the cylindrical surface 
(process action 1202). This is repeated for each pixel loca- 
tion on the cylindrical surface. For example, this metric 
could be the variance in some prescribed pixel characteristic, 
such as pixel intensity or color. It is also preferred (although 
optional) that the result of the metric employed be spatially 
aggregated at each pixel location of each cylindrical surface 
using an appropriate convolution technique (process action 
1204). For example, a moving average box filter technique 
could be used, or the other techniques, such as the technique 
described in reference [17]. 

The result of the metric (once spatially aggregated if 
desired) is essentially a cost function. The theory behind the 
cylindrical sweep approach is that the pixels in each multi- 
perspective panorama, which depict the same portion of the 
scene, will converge to approximately the same location on 
one of the candidate cylindrical surfaces. Thus, the pixel 
location on a cylindrical surface having the lowest cost 
function in comparison to the cost function computed for the 
corresponding location on the other surfaces (i.e., the "win- 
ning" location) should correspond to the actual position of 
the portion of the scene depicted by the pixels projected 
from the multiperspective panoramas to that location. It is 
noted that not all the pixels projected from the multiper- 
spective panoramas to the "winning" location may depict the 
same portion of the scene due to possible occlusions in some 
of the panoramas employed. It is further noted that by 
analyzing the cost function data starting with the cylindrical 
surface having the smallest diameter and working out from 
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there, it is possible to determine any occlusion relationships 
depicted in the multiperspective panoramas via conventional 
methods [18]. In view of the above, the next process action 
1206 is to determine, for each corresponding pixel location 
of the cylindrical surfaces, the lowest cost function associ- 
ated with that pixel location among all the cylindrical 
surfaces. For each "winning" location, the panoramic coor- 
dinates are identified and assigned as the actual position of 
the portion of the scene depicted by the pixels projected 
from the multiperspective panoramas to that location 
(process action 1208). The panoramic view coordinates are 
defined by the vertical height of the pixel location on the 
cylindrical surface, the rotation angle with respect to the 
center of rotation and the radius of the cylindrical surface 
from the center of rotation. 

In the foregoing way, the actual position in panoramic 
coordinates of the portion of the scene depicted in the pixels 
of the multiperspective panoramas can be determined. Based 
on this information, a depth of the scene can be computed in 
the conventional manner. Essentially, the depth map is a 
panoramic image of the scene where the distances or depths 
from the center of rotation associated with the multiperspec- 
tive panoramas to the portion of the scene depicted by each 
pixel is represented by a particular intensity value. 

While the foregoing implementation of the cylindrical 
sweep approach represents one preferred example, it is noted 
that other variations are possible and may be employed as 
desired. Loosely speaking, traditional methods of computing 
dense stereo depth can be divided into two classes. The first 
class of methods allow every pixel to independently select 
its disparity. These methods typically use statistical methods 
and are usually based on a correlation of some type. The 
above-described implementation of the cylindrical sweep 
process employs one such correlation technique to deter- 
mine the winning pixel locations. However, other existing 
correlation-based methods could be used instead, as desired. 
In addition, the aforementioned second class of methods 
could be used to determine the winning pixel locations. 
Essentially, this second class of methods finds the depth map 
that minimizes some function associated with the pixels, 
usually called the energy or the objective function, using 
some optimization technique. In other words, the best set of 
depth values are simultaneous found by minimizing a global 
cost criterion. These methods are generally referred to as 
global optimization or global regularization techniques. It is 
envisioned that global optimization techniques could also be 
employed to determine the winning pixel locations, rather 
than the correlation-based method (or variation thereof) 
described in the above example. The exact energy function 
and optimization method employed will depend on the 
application, but can be selected from any of the current 
global energy minimization techniques. 
2.2 Generating 3D Reconstructions Using Traditional Stereo 
Matching Algorithms When an Approximately Horizontal 
Epipolar Geometry Exists Between Multiperspective Pan- 
oramas 

It was mentioned earlier that if conditions are right, 
traditional stereo matching algorithms can also be employed 
without modification to compute the desired 3D reconstruc- 
tion (and so a depth map) of the scene. This would have 
advantages as these types of algorithms can be less compu- 
tationally intensive than the previously-described cylindrical 
sweeping approach. 

One example of the right conditions involves the use of 
two of the multiperspectives that constitute symmetric pairs. 
A symmetric pair of multiperspective panoramas exists 
where, referring once again to FIG. 11, the optical rays used 
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to construct two multiperspective panoramas are symmetric 
with respect to the swing line CV, i.e., fam- $ 2 (also assum- 
ing that il)f=4,- or This occurs when, for example, the 
left and right columns of a swing stereo sequence are 

5 selected or when two oppositely facing cameras are fixed at 
equal distances but at opposite ends of a rotating boom 
(concentric panorama). For a symmetric pair of concentric 
multiperspective panoramas, the epipolar geometry consists 
of horizontal lines. This follows directly from equation (4), 

10 i.e., S2 :1 «l. A more informal proof could be obtained by 
drawing another point P' in FIG. 11 at an angle -<j>, and 
observing that z*=d cos is the same for both viewing rays. 
A direct consequence of selecting such a pair of panoramic 
images is that any traditional stereo algorithm (e.g., hierar- 

15 chical warp [15, 2]) can be used, to compute the desired 
reconstruction. 

However, in practice it is desirable to use more than two 
images, in order to obtain a more accurate and robust 
correspondence [12, 3], but it is also desirable to avoid 

20 having to be restricted to the previously described cylindri- 
cal sweep approach for computing depth maps. In this 
section, it will be discussed whether the epipolar geometry 
is sufficiently close to a horizontal epipolar geometry so that 
other traditional multi-image stereo matching algorithms 

25 (such as SSSD [12]) could be used instead. A requirement 
for a traditional multi-baseline stereo algorithm to be appli- 
cable is that the location of pixels at different depths can be 
explained by a collection of pinhole cameras. In the present 
case, a further requirement is that horizontal epipolar geom- 

30 etry apply, in which case the horizontal parallax would be of 
the form 

AB k . 0 =mJ{r), 

35 i.e., that a fixed linear relationship exists between horizontal 
parallax and some common function of r for all images. The 
horizontal parallax equation given in (3) does not exactly 
satisfy this requirement. However, if either R/r or sin <J> are 
small in both images, then 

40 MuARt sin <t>t-«o sin ^y 1 (6) 

It can therefore be seen that inverse r plays the same role as 
inverse depth (disparity [12]) does in multi-baseline stereo. 
FIGS. 13 and 14 plot the exact formula (3) for horizontal 

45 parallax as a function of r" 1 for various values of $ and R. 
The plot of FIG. 13 shows the ratio of A0 fcO to 1/r (since it 
is very hard to tell the deviation from linearity by eye) for 
<(> 0 =0 (central column) and varying $ fc (swing panoramas). 
The plot of FIG. 14 shows the ratio of A8* :0 to 1/r for R o «0 

50 (no parallax) and varying R* for concentric panoramas with 
^=90° and tJ>*=0. As can be seen from these plots, the linear 
approximation to parallax is quite good, as long as the 
nearest scene point doesn't get too close, e.g., no closer than 
50% of R for moderate focal lengths. The reduced linearity 

55 in the concentric stereo rig can be mitigated by spacing the 
cameras more closely, i.e., not using the full baseline of the 
arm if scene points are too close. 

A second requirement of assuming a horizontal geometry 
is that the vertical parallax needs to be negligible (preferably 

60 under one pixel). For images of about 240 lines (a single 
field from NTSC video), it is desired that |Ay|^ 1201^ 2 - 
1|<1 (120 is the half-height of the image), i.e., (s^-l 
|<0.008. 

The vertical parallax equation (4) can be approximated 
65 under two different conditions. For swing stereo (Ri es R 2 » 
ijj-(t>), assume that <|> 2 «0 (central column) and fa is small. If 
so equation (4) can be expanded to obtain 
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employ two of them. However, it is preferred that as many 

IR 2 , R R . , ]r /?i C7) of the panoramas as possible be used to generate the 3D 

1 - ^ siu 2 d, - - + — svffa 1 + - r . r . _ & „ ... 

r 2r ji r J reconstruction to improve its accuracy. Regardless, as mdi- 

A cated by process action 1714, if only two multiperspective 

* cos ^ 1 1 1 + 2r sm ^ 1 J 5 panoramas are used, a traditional stereo matching algorithm 

designed for a pair of images is used to generate a 3D 

_ ...... , , . • , - reconstruction of the scene. However, if more than two 

Thus, once the global scale change by cos ifc (which is mu i ti perspective panoramas are employed, then a so-called 

independent of depth) is compensated for, there » a vertical multi . image stereo matching process is ^ to CTeat e the 

parallax component that is linear m R/r and quadratic in sin 10 desire<J re ^ nstn|Ctioil of m * £ ene (process action 1716) 

For concentric panoramas, <|)-90 o and o[)=0, which means 3.0 Experiments 

It has been shown experimentally that good stereo recon- 

Ji-R 2 /^ structions can be obtained from multiperspective 

j 2: i = 1 = l + —{R\ - R]). 15 panoramas, and that the original parallax in the scene can be 

- /rl/r 2 recreated from just one panorama and one panoramic depth 

map. It is also possible to extrapolate novel views from 

_ . , „ . . i • . original panoramas and the recovered depth map. 

The vertical parallax is inversely proportional to squared .„ „ , , 

distance r, and proportional to the difference in squared radii c Specifically, we have apphed our stereo reconstruction 

R. FIGS. 15 and 16 plot the exact formula (4) for vertical 20 multiperspective panoramas system and process to 

parallax as a function of r 1 for various values of <(> and R. both synthetic data and real data. For example, as indicated 

The plot of FIG. 15 shows scale change s M -l for <|> 0 =0 previously, FIGS. 5(a) and (b) depict two synthesized con- 

(central column) and varying (swing panoramas). The centric panoramas. In all we synthesized seven (7) such 

plot of FIG. 16 shows s fc0 -l for R o «0 (no parallax) and panoramas, each representing what a slit camera would have 

varying R* (concentric panoramas with <jy-90° and uj*-0). 25 produced if rotated around circles of different radii (i.e., 0.4, 

As can be seen from these plots, the amounts of vertical °- 5 » ■ ■ • 10 )- FIG * 5 ( a ) represents the concentric panoramas 

parallax are quite small (<1%) if the field of view is associated with the outermost camera position (i.e., R-1.0) 

moderately small (say about 30°) or the ratio of the nearest arid FIG - 5 ( b ) represents the concentric panorama associated 

scene point to the variation in radial camera positions is wi* the middle camera position (i.e., R=0.7). FIG. 5(c) 

j ar g e 30 shows the estimated depth map associated with a panorama 

In view of the foregoing description, it can be stated that at the middle camera position, which was derived from the 

when the ratio R/r and/or off-axis angle <|> are small, a nearly synthesized concentric panoramas. It should be noted 

horizontal epipolar geometry (i.e., classic multi-baseline that the concentric panoramas were synthesized using a 

stereo geometry [12]) is obtained, after compensating once relatively small specified horizontal field of view (i.e., 24°). 

for the vertical scaling of each image. Thus, if these condi- 35 Because of this small field of view, the panoramas exhibit 

tions exist, or if two symmetric multiperspective panoramas negligible vertical parallax. Using the estimated depth map 

are being used, then traditional stereo matching algorithms shown m F 10 * 5 ( c )> we synthesize the concentric panorama 

can be employed for the stereo reconstruction process. SD0Wn in 5 (<0 with me same camera parameters as in 5(a). 

Otherwise, the previously-described cylindrical sweeping ^ new panorama is almost indistinguishable from S(a) 

process should be employed for this purpose. More 40 except in the regions where significant occlusion occurs as 

specifically, referring to FIGS. 17A and 17B, the multiper- shown in the close-up in 5(h). Notice that the spotlight is 

spective panoramas are first constructed via any desired synthesized well even though its depth estimation is clearly 

method (process action 1700). It is then decided if the stereo wrong. 

reconstruction process is to be conducted using just two Referring now to FIGS. 10(a) and (6), two swing pan- 
symmetric multiperspective panoramas as indicated by pro- 45 oramas respectively created using the leftmost column and 
cess action 1702. If not, then in process action 1704, it is the center column from each single-perspective image cap- 
determined whether R/r is small (e.g., less than about 0.7) tured during a real swing sequence of a lab scene. We used 
for each of the multiperspective panoramas being used. If a digital video camera in portrait mode with a field of view 
this ratio is not small enough, then in process action 1706, of around 27° by 36° and a digitized image size of 240 by 
it is determined whether the off-axis angle <(> is small (e.g., 50 320 pixels. The camera was mounted off-centered on a plate 
less than about 15°) for each of the multiperspective pan- rotated with a step motor which provided accurate rotation 
oramas being used. If the off-axis angle is not small enough, parameters. After scaling the images vertically by cos i|>, we 
then in process action 1708, the cylindrical sweep process is found that there was a small 0.5 pixel drift remaining 
used to create a 3D reconstruction of the scene. between the panoramas. This was probably caused by a 
If, however, a symmetric pair of multiperspective pan- 55 slight rotation of the camera around its optical axis. In 
oramas is to be employed, or R/r is small enough, or <(> is practice, compensating for such registration errors is not 
small enough, then traditional stereo matching algorithms difficult: a simple point tracker followed by linear regression 
are employed for the reconstruction process. Specifically, can be used to obtain the best possible horizontal epipolar 
any global scale difference (as described by equation (4)) is geometry. The panoramas after vertical scaling and drift 
first compensated for in each multiperspective panorama 60 compensation are shown in the close-up regions in FIGS, 
(process action 1710). Then, as indicated by process action 10(e) through 10(g). As you can see, very little vertical 
1712, it is decided if more than two multiperspective pan- parallax remains. The reconstructed depth map generated 
oramas are to be used create a 3D reconstruction of the scene from the swing panoramas formed from the single- 
from the panoramas. In the case where just two symmetric perspective images of the lab scene is shown in FIG. 10(c), 
panoramas are being used, the answer is clearly no. 65 and a synthesized novel panorama (from an extrapolated 
However, in cases where more than two multiperspective viewpoint) and its close-up are shown in FIGS. 10(d) and 
panoramas have been constructed, the option exist to just 10(f), respectively. 
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4.0 Alternate Techniques [7] S. B. Kang and R Szeliski. 3-D scene data recovery using 

While the invention has been described in detail by omni-directional multibaseline stereo. International Jour- 

specific reference to preferred embodiments thereof, it is nal of Computer Vision, 25 (2): 167-183, November 

understood that variations and modifications thereof may be 1997. 

made without departing from the true spirit and scope of the 5 P] A- Krishnan and N. Ahuja. Range estimation from focus 

invention. For example, while it has been assumed that the ^8 a non-frontal imaging camera. International Jour- 

cameras used to capture the single-perspective images °f Computer Vision, 20 (3): 169-185, March/Apnl 

employed to form the multiperspective panoramas in accor- r ^??^\ t , , ^ m , ^ . 

i r M „ ... t , - ,|i mni ' „ • tUa M „ [9] K. N. Kutulakos and S. M. Seitz. A theory of shape by 

dance with the present invention, are all moving in the same LJ . ^ * M , T - • r 

. *t_ • * ii *u c . n space carving. C S Technical Report 692, University of 

plane, there is actually nothing preventing the cameras from 10 R H ochester Rochester N Y. May 1998 

being located in different (parallel) planes [19], so long as [1Q] M Uv * and p Hanrahan. Light field rendering. In 

their rotation axes are all the same (i.e^the camera motions Computer Graphics Proceedings, Annual Conference 

are co-axial, rather than concentric). The only difference in SerieSf pages 31 _^ 2 , Proc. SIGGRAPH'96 (New 

this case is the addition of some extra depth-dependent Orleans), August 1996. ACM SIG-GRAPH. 

vertical parallax, which can easily be accommodated in both is [n] j_ McMillan and G. Bishop. Plenoptic modeling: An 

traditional stereo algorithms and in the previously-described image-based rendering system. Computer Graphics 

cylindrical sweep process using additional vertical scaling (SIGGRAPH'95),pages 39-46, August 1995. 

and shifting factors. Further, it is noted that a more general [12] M. Okutomi andT. Kanade. A multiple baseline stereo, 

camera configuration can be used to create the swing pan- IEEE Transactions on Pattern Analysis and Machine 

oramas according to the present invention where one or 20 Intelligence, 15 (4): 353-363, April 1993. 

more cameras are angled at various orientations with respect [13] R. Peleg and M. Ben-Ezra. Stereo panorama with a 

to the swing arm and one or more columns are sampled from single camera. In CVPK '99, pages 395-401, Fort Collins, 

each camera. Finally, while the camera path used to capture June 1999. 

the single-perspective images from which the multiperspec- [14] S. Peleg and J. Herman. Panoramic mosaics by mani- 

tive panoramas were derived have been described as being 25 fold projection. In IEEE Computer Society Conference on 

circular, this need not be the case. The path that the camera Computer Vision and Pattern Recognition (CVPR>97), 

takes can be any continuous path around a center of rotation. pages 338-343 San Juan Puerto Rico, June 1997 

However, if the multiperspective panorama is to be con- j Qu^m^Hierarchical wa^p stereo. In Image Under- 

structed in a manner similar to a concentric panorama, then \^JmL l°984. lien^Applications limited Cc^ 

each camera path should be identical but proportionally 30 oora ti on 

larger or smaller than an adjacent path. The process for [16] p Rademacher and B ishop G. Multiple -center-of- 

generating the 3D reconstruction of the scene remains the - projection images. In Computer Graphics Proceedings, 

same, regardless of the shape of the path. The use of Annml Conference Series, pages 199-206, Proc. SIG- 

non-circular paths may be advantageous under certain cir- GRAPH'98 (Orlando), July 1998. ACMSIGGRAPH. 

cumstances. For example, a non-circular path might be 35 [17] d. Scharstein and R. Szeliski. Stereo matching with 

better suited to keeping the R/r ratio small at every rotational non-linear diffusion. International Journal of Computer 

angle 6 around the centroid of the path, thereby allowing the Vision, 28 (2): 155-174, July 1998 

use of traditional stereo reconstruction algorithms. [18] S. M. Seitz and C. M. Dyer, Photorealistic scene 

REFERENCES reconstrcution by space coloring. In IEEE Computer 

40 Society Conference on Computer Vision and Pattern 

[1] S. Baker, R. Szeliski, and P. Anandan. A layered Recognition (CVPR'97), pages 1067-1073, San Juan, 

approach to stereo reconstruction. In IEEE Computer Puerto Rico, June 1997. 

Society Conference on Computer Vision and Pattern [19] H.-Y. Shum et al. Omnivergent stereo. In Seventh 

Recognition (CVPK '98), pages 434-441, Santa Barbara, International Conference on Computer Vision {ICC V'99), 

June 1998. 45 Greece, September 1999. 

[2] J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani. [20] H.-Y. Shum and L.-W. He. Rendering with concentric 

Hierarchical model-based motion estimation. In Second mosaics. In SIGGRAPH* 99, Los Angeles, August 1997. 

European Conference on Computer Vision (ECCV'92), [21] R. Szeliski. A multi-view approach to motion and 

pages 237-252, Santa Margherita Liguere, Italy, May stereo. In IEEE Comp. Soc. Conf. On Computer Vision 

1992. Springer- Verlag. 50 and Pattern Recognition (CVPR '99), pages 157-163, Fort 

[3] R. T. Collins. A space-sweep approach to true multi- Collins, June 1999. 

image matching. In IEEE Computer Society Conference [22] R. Szeliski and P. Golland. Stereo matching with 

on Computer Vision and Pattern Recognition (CVPR'96), transparency and matting. In Sixth International Confer- 

pages 358-363, San Francisco, Calif., June 1996. ence on Computer Vision (ICCV'98), pages 517-524, 
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Cohen. The lumigraph. In Computer Graphics [23] R. Szeliski and H.-Y. Shum. Creating full view pan- 
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Proc. SIGGRAPH'96 (New Orleans), August 1996. ACM puter Graphics (SIGGRAPH'91), pages 251-258, August 

SIGGRAPH. 1997. 

[5] R. Gupta and R. I. Hartley. Linear pushbroom cameras. 60 [24] D. N. Wood et al. Multiperspective panoramas for eel 

IEEE Transactions on Pattern Analysis and Machine animation. In Computer Graphics Proceedings, Annual 

Intelligence, 19 (9): 963-975, September 1997. Conference Series, pages 243-250, Proc. SIGGRAPH'97 

[6] T. Kanade et al. A stereo machine for video-rate dense (Los Angeles), August 1997. ACM SIGGRAPH. 

depth mapping and its new applications. In IEEE Com- Wherefore, what is claimed is: 

puter Society Conference on Computer Vision and Pattern 65 1. A computer-implemented process for computing a 3D 

Recognition (CVPR'96), pages 96-202, San Francisco, reconstruction of a scene, comprising using a computer to 

Calif., June 1996. perform the following process actions: 
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constructing at least two multiperspective panoramas 
from a collection of single-perspective images of the 
scene each of which was captured from a different, but 
co-axial, viewpoint, wherein each multiperspective 
panorama comprises a plurality of side-by-side colum- 
nar images depicting consecutive portions of the scene 
each of which was obtained from a different one of the 
single-perspective images; and 

computing the 3D reconstruction of the scene from the at 
least two multiperspective panoramas. 

2. The process of claim 1, wherein the process action of 
constructing at least two multiperspective panoramas com- 
prises an act of, for each multiperspective panorama, using 
columnar images obtained from single-perspective images 
whose viewpoints are all at the same fixed radius from a 
center of rotation and which are all laterally centered about 
a line forming a particular off-axis angle with respect to a 
swing line, said swing line being defined by the center of 
rotation and the viewpoint associated with each single - 
perspective images from which the respective columnar 
images were obtained. 

3. The process of claim 2, wherein the off-axis angle is 90 
degrees for each columnar image, and the magnitude of the 
fixed radius is different for each multiperspective panorama. 

4. The process of claim 2, wherein the fixed radius is the 
same for each multiperspective panorama and the off-axis 
angle is different for each multiperspective panorama. 

5. The process of claim 2, wherein the viewpoints asso- 
ciated with all the columnar images are co -planar. 

6. The process of claim 1, wherein the process action of 
computing a 3D reconstruction of the scene comprises the 
acts of: - . _ 

projecting each pixel of each multiperspective panorama 
being used to compute the 3D reconstruction onto each 
of a series of cylindrical surfaces of progressively 
increasing radii, all of which exceed the radius of the 
outermost multiperspective panorama employed; 

for each pixel location on each cylindrical surface, com- 
puting a fitness metric for all the pixels projected from 
each multiperspective panorama onto the pixel 
location, wherein said fitness metric provides an indi- 
cation as to how closely a prescribed characteristic of 
said projected pixels match each other; 

for each respective group of corresponding pixel locations 
of the cylindrical surfaces, determining which particu- 
lar location of the group has a computed fitness metric 
that indicates the prescribed characteristic of said pro- 
jected pixels matches more closely than the rest 
(hereinafter referred to as the winning pixel location), 
wherein a group of corresponding pixel locations of the 
cylindrical surfaces are those which depict the same 
portion of the scene; and 

for each winning pixel location, identifying its panoramic 
coordinates and designating these coordinates to be the 
position of the portion of the scene depicted by the 
pixels projected from the multiperspective panoramas 
to that location. 

7. The process of claim 6, wherein the process action of 
determining the winning pixel location comprises an act of 
employing a correlation-based stereo depth technique to 
identify which particular location of each group of corre- 
sponding pixel locations of the cylindrical surfaces has a 
computed fitness metric that indicates the prescribed char- 
acteristic of said projected pixels matches more closely than 
the rest. 

8. The process of claim 6, wherein the process action of 
determining the winning pixel location comprises an act of 
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employing a global optimization stereo depth technique to 
identify which particular location of each group of corre- 
sponding pixel locations of the cylindrical surfaces has a 
computed fitness metric that indicates the prescribed char- 
acteristic of said projected pixels matches more closely than 
the rest. 

9. The process of claim 6, wherein the process action of 
computing a 3D reconstruction of the scene further com- 
prises the act of spatially aggregating the computed fitness 
metric at each pixel location on each cylindrical surface, 
prior to performing the process action of determining the 
winning pixel location for each respective group of corre- 
sponding pixel locations of the cylindrical surfaces. 

10. The process of claim 1, wherein the process action of 
constructing at least two multiperspective panoramas com- 
prises an act of constructing a pair of symmetric multiper- 
spective panoramas, wherein two multiperspective panora- 
mas are symmetric whenever, 

the columnar images used to construct each panorama are 
all obtained from single perspective images whose 
viewpoints are at the same radius from a common 
center of rotation, 

the columnar images used to construct the first panorama 
of the pair are all centered about a first line forming an 
off-axis angle in the clockwise direction with respect to 
a swing line, said swing line being defined by the 
common center of rotation and the viewpoint associ- 
ated with each single-perspective images from which 
the respective columnar images were obtained, and 

the columnar images used to construct the second pan- 
orama of the pair are all centered about a second line 
forming the same off-axis angle with respect to the 
swing line but in the counterclockwise direction. 

11. The process of claim 10, wherein the process action of 
computing a 3D reconstruction of the scene comprises the 
acts of: 

employing the pair of symmetric multiperspective pan- 
oramas; 

compensating for any global vertical scale difference in 
each multiperspective panorama; and 

using any traditional stereo matching algorithm requiring 
a horizontal epipolar geometry to designate the coor- 
dinates of the portion of the scene depicted by each 
corresponding pixel location of the multiperspective 
panoramas. 

12. The process of claim 1, wherein the process action of 
constructing at least two multiperspective panoramas com- 
prises an act of constructing each multiperspective pan- 
orama using columnar images obtained from single perspec- 
tive images whose viewpoints are all at the same radius from 
a center of rotation, and wherein said radius is less than 
approximately 0.7 the distance from the center of rotation 
and the nearest point in the scene depicted in any of the 
columnar images. 

13. The process of claim 12, wherein the process action of 
computing a 3D reconstruction of the scene comprises the 
acts of: 

compensating for any global vertical scale difference in 
each multiperspective panorama; and 

using any traditional stereo matching algorithm requiring 
a horizontal epipolar geometry to designate the coor- 
dinates of the portion of the scene depicted by each 
corresponding pixel location of the multiperspective 
panoramas. 

14. The process of claim 13, wherein there are more than 
two multiperspective panoramas used to compute the 3D 
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reconstruction of the scene, and wherein the traditional 
stereo matching algorithm is a multi-image stereo matching 
process. 

15. The process of claim 1, wherein the process action of 
constructing at least two multiperspective panoramas com- 
prises an act of constructing each multiperspective pan- 
orama using columnar images obtained from single perspec- 
tive images whose viewpoints are all at the same fixed radius 
from a center of rotation and which are all laterally centered 
about a line forming an off-axis angle with respect to a swing 
line, said swing line being defined by the center of rotation 
and the viewpoint associated with each single-perspective 
images from which the respective columnar images were 
obtained, and wherein the off-axis angle does not exceed 
approximately 15 degrees. 

16. The process of claim 15, wherein the process action of 15 
computing a 3D reconstruction of the scene comprises the 
acts of: 

compensating for any global vertical scale difference in 
each multiperspective panorama; and 

using any traditional stereo matching algorithm requiring 
a horizontal epipolar geometry to designate the coor- 
dinates of the portion of the scene depicted by each 
corresponding pixel location of the multiperspective 
panoramas. 

17. The process of claim 16, wherein there are more than 
two multiperspective panoramas used to compute the 3D 
reconstruction of the scene, and wherein the traditional 
stereo matching algorithm is a multi-image stereo matching 
process. 

18. The process of claim 1, further comprising the act of 
creating a depth map of the scene from the computed 3D 
reconstruction. - - - 

19. A system for computing a 3D reconstruction of a 
scene, comprising: 

a general purpose computing device; 

a computer program comprising program modules 
executable by the computing device, wherein the com- 
puting device is directed by the program modules of the 
computer program to, 

construct at least two multiperspective panoramas from 
a collection of single-perspective images of the scene 
each of which was captured from a different, but 
co-axial, viewpoint, wherein each multiperspective 
panorama comprises a plurality of side-by-side 
columnar images depicting consecutive portions of 45 
the scene each of which was obtained from a differ- 
ent one of the single-perspective images, and 

compute the 3D reconstruction of the scene from the at 
least two multiperspective panoramas. 

20. The system of claim 19, wherein the program module 
for constructing at least two multiperspective panoramas 
comprises a sub-module for employing at least one camera 
to capture the single -perspective images of the scene, said 
camera's motion being constrained to a radial path around a 
fixed rotation center, and wherein each single-perspective 
image is captured at one of a series of consecutive rotation 
angles around the center of rotation. 

21. The system of claim 20, wherein the camera is a slit 
image camera, and wherein each columnar image is a slit 
image of the scene captured by said camera. 

22. The system of claim 21, wherein said slit image is a 
one pixel wide column. 

23. The system of claim 20, wherein the camera is a 
perspective view camera, and wherein each columnar image 
comprises a portion of the image captured by the camera. 

24. The system of claim 23, wherein said portion of the 
image captured by the camera is a one pixel wide column. 
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25. The system of claim 19, wherein the program module 
for constructing at least two multiperspective panoramas 
comprises a sub-module for extracting the single- 
perspective images from a synthesizes scene, wherein each 
extracted image depicts a part of the synthesized scene that 
a camera would capture if its motion were constrained to a 
radial path around a fixed rotation center and each image 
was captured at one of a series of consecutive rotation angles 
around the center of rotation. 

26. The system of claim 25, wherein images extracted are 
one-pixel wide. 

27. The system of claim 19, wherein the program module 
for constructing at least two multiperspective panoramas 
comprises, for each multiperspective panorama, using 
columnar images obtained from single-perspective images 
whose viewpoints are all at the same fixed radius from a 
center of rotation and which are all laterally centered about 
a line forming a particular off-axis angle with respect to a 
swing line, said swing line being defined by the center of 
rotation and the viewpoint associated with each single - 
perspective images from which the respective columnar 
images were obtained. 

28. The system of claim 27, wherein the off-axis angle is 
90 degrees for each columnar image, and the magnitude of 
the fixed radius is different for each multiperspective pan- 
orama. 

29. The system of claim 27, wherein the fixed radius is the 
same for each multiperspective panorama and the off-axis 
angle is different for each multiperspective panorama. 

30. The system of claim 27, wherein the viewpoints 
associated with all the columnar images are co-planar. 

31. The system of claim 19, wherein the program module 
for computing a 3D reconstruction of the scene comprises 
sub-modules for: 

projecting each pixel of each multiperspective panorama 
being used to compute the 3D reconstruction onto each 
of a series of cylindrical surfaces of progressively 
increasing radii, all of which exceed the radius of the 
outermost multiperspective panorama employed; 

for each pixel location on each cylindrical surface, com- 
puting a fitness metric for all the pixels projected from 
each multiperspective panorama onto the pixel 
location, wherein said fitness metric provides an indi- 
cation as to how closely a prescribed characteristic of 
said projected pixels match each other; 

for each respective group of corresponding pixel locations 
of the cylindrical surfaces, determining which particu- 
lar location of the group has a computed fitness metric 
that indicates the prescribed characteristic of said pro- 
jected pixels matches more closely than the rest 
(hereinafter referred to as the winning pixel location), 
wherein a group of corresponding pixel locations of the 
cylindrical surfaces are those which depict the same 
portion of the scene; and 

for each winning pixel location, identifying its panoramic 
coordinates and designating these coordinates to be the 
position of the portion of the scene depicted by the 
pixels projected from the multiperspective panoramas 
to that location. 

32. The system of claim 31, wherein the sub-module for 
determining the winning pixel location comprises a sub- 
module for employing a correlation-based stereo depth 
technique to identify which particular location of each group 
of corresponding pixel locations of the cylindrical surfaces 
has a computed fitness metric that indicates the prescribed 
characteristic of said projected pixels matches more closely 
than the rest. 
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33. The system of claim 31, wherein the sub-module for 39. The system of claim 38, wherein there are more than 
determining the winning pixel location comprises a sub- two multiperspective panoramas used to compute the 3D 
module for employing a global optimization stereo depth reconstruction of the scene, and wherein the traditional 
technique to identify which particular location of each group stereo matching algorithm is a multi-image stereo matching 
of corresponding pixel locations of the cylindrical surfaces 5 process. 

has a computed fitness metric that indicates the prescribed 40 ^ system of claim 19, wherein the program module 

characteristic of said projected pixels matches more closely for constructing at least two multiperspective panoramas 

than the rest comprises a sub-module for constructing each multiperspec- 

34. The system of claim 31, wherein the program module tive P™ 3 usin 8 c °«™™ ima 8 es obtained from single 
for computing a 3D reconstruction of the scene further 10 Wctive images whose viewpoints are all at the same 
comprises a sub-module for spatially aggregating the com- ? xed from a cem , er °/ rotatlon and wmch f* M 
puted fitness metric at each pixel location on each cylindri- lateraU y cen,ered about a ^ formm B an angle with 
cal surface, prior to performing the process action of deter- r6S P 6ct to a swm S sald swm 8 hne bem § defined by the 
mining the winning pixel location for each respective group cent f r of mmion and thev,ewpoint associated with each 
of corresponding pixel locations of the cylindrical surfaces, is single-perspective linages from which the respective colum- 

35. The system of claim 19, wherein the program module nar m,a 6 es wel * ° bta "ied, and wherein the off-axis angle 
for constructing at least two multiperspective panoramas d «* n °? exceed approximately 15 degrees. 

comprises a sub-module for constructing a pair of symmetric «. The system of claim 40, wherein the program module 

multiperspective panoramas, wherein two multiperspective for _ computing a 3D reconstruction of the scene comprises 

♦ • u sub-modules for: 

panoramas are symmetric whenever, 20 

t - j , . . . compensating for any global vertical scale difference in 

the columnar images used to construct each panorama are r , . P & , 

11 i_. * j c - 1 4- - u each multiperspective panorama; and 

all obtained from smgle perspective images whose r r r ' 

viewpoints are at the same radius from a common using any traditional stereo matching algorithm requiring 

center of rotation a nonzonta l epipolar geometry to designate the coor- 

. . , 4 lL c t 9 c dinates of the portion of the scene depicted by each 

the columnar unages used to construct the first panorama . , , - r lA . J t . 

- , . , . 4 ~ t ,. c r . correspondmg pixel location 01 the multiperspective 

of the pair are all centered about a first line forming an r 0 r r r 

off-axis angle in the clockwise direction with respect to .J^ 0 ™™^" c , - A ~ , - it _ 

,P . « . , - , * iL 42. The system of claim 41, wherein there are more than 

a swine line, said swing line being defined by the n- *• j * * lL ^ 

& ' c 4 t . * ,1 - . t J . two multiperspective panoramas used to compute the 3D 

common center of rotation and the viewpoint associ- * r *u -1 u • *u . j*** 1 

... ... ... r r- L - L in reconstruction of the scene, and wherein the traditional 

ated with each smgle -perspective images from which 4 4 , . w . . . t . . 

. . t ^ r . r . . « j stereo matching algorithm is a multi- image stereo matching 

the respective columnar images were obtained, and 0 0 0 0 

the columnar images used to construct the second pan- system of claim 19 , comprising a program 

orama of the pair are all centered about a second lme moduk for creating a depth map of the from the 

forming the same off-axis angle with respect to the ^ computed 3D reconstruction. 

swing line but in the counterclockwise direction. 44 A computer . 1BadaWe memory for computing a 3D 

36. The system of claim 35, wherein the program module reconstruction of a scene, comprising: 

for computing a 3D reconstruction of the scene comprises & uter . readable ^ medium; ^ 

sub-modules for: . . . , . 

, . t _ . . . a computer program composing program modules stored 

employing the pair of symmetric multiperspective pan- ^ m me storage medium wnerein ^ storage medjum is 

oramas, ^ configured by the computer program that it causes a 

compensating for any global vertical scale difference in computer to, 

each multiperspective panorama; and construct at least two multiperspective panoramas from 

using any traditional stereo matching algorithm requiring a collection of single-perspective images of the scene 

a horizontal epipolar geometry to designate the coor- 45 eacn 0 f wn ich was captured from a different, but 

dinates of the portion of the scene depicted by each co-axial, viewpoint, wherein each multiperspective 

corresponding pixel location of the multiperspective panorama comprises a plurality of side-by-side 

panoramas. columnar images depicting consecutive portions of 

37. The system of claim 19, wherein the program module tne scene each of which was obtained from a differ- 
for constructing at least two multiperspective panoramas 50 ent one of the single-perspective images, and 
comprises a sub-module for constructing each multiperspec- compute the 3D reconstruction of the scene from the at 
tive panorama using columnar images obtained from single i eas t two multiperspective panoramas, 
perspective images whose viewpoints are all at the same 45. The computer-readable memory of claim 44, wherein 
radius from a center of rotation, and wherein said radius is the program module for constructing at least two multiper- 
less than approximately 0.7 of the distance from the center 55 spective panoramas comprises, for each multiperspective 
of rotation and the nearest point in the scene depicted in any panorama, using columnar images obtained from single - 
of the columnar images. perspective images whose viewpoints are all at the same 

38. The system of claim 37, wherein the program module fi xec j ra di us from a center of rotation and which are all 
for computing a 3D reconstruction of the scene comprises laterally centered about a line forming a particular off- axis 
sub-modules for: 60 angle with respect to a swing line, said swing line being 

compensating for any global vertical scale difference in defined by the center of rotation and the viewpoint associ- 

each multiperspective panorama; and ated with each single-perspective images from which the 

using any traditional stereo matching algorithm requiring respective columnar images were obtained. 

a horizontal epipolar geometry to designate the coor- 46. The computer-readable memory of claim 45, wherein 

dinates of the portion of the scene depicted by each 65 the off-axis angle is 90 degrees for each columnar image, 

corresponding pixel location of the multiperspective and the magnitude of the fixed radius is different for each 

panoramas. multiperspective panorama. 
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47. The computer-readable memory of claim 45, wherein viewpoints are at the same radius from a common 
the fixed radius is the same for each multiperspective center of rotation, 

panorama and the off-axis angle is different for each mul- . . , t 4 , 4 . „ t 

f. . ■ „ M ^ the columnar images used to construct the first panorama 

tiperspective panorama. * * 

48. The computer-readable memory of claim 45, wherein 5 of the P air are aU centered about a first line forming an 
the viewpoints associated with all the columnar images are off-axis angle in the clockwise direction with respect to 
co-planar. a swing line, said swing line being defined by the 

49. The computer-readable memory of claim 44, wherein common center of rotation and the viewpoint associ- 
the program module for computing a 3D reconstruction of ated ^ eacn single-perspective images from which 
the scene comprises sub-modules for: 1Q ^ ^ cohimnar ^ were obtained> aad 

projecting each pixel of each multiperspective panorama 

being used to compute the 3D reconstruction onto each the columnar images used to construct the second pan- 

of a series of cylindrical surfaces of progressively orama of the pair are all centered about a second line 

increasing radii, all of which exceed the radius of the forming the same off-axis angle with respect to the 

outermost multiperspective panorama employed; 5 swing line but in the counterclockwise direction. 

for each pixel location on each cylindrical surface, com- 54. The computer-readable memory of claim 53, wherein 

puting a fitness metric for all the pixels projected from the program module for computing a 3D reconstruction of 

each multiperspective panorama onto the pixel the scene comprises sub-modules for: 

location, wherein said fitness metric provides an indi- , , u c . . u . 

., / . . c employing the pair of symmetric multiperspective pan- 
cation as to how closely a prescribed characteristic or ~ n ' 

oram as* 

said projected pixels match each other; ' 

for each respective group of corresponding pixel locations compensating for any global vertical scale difference in 

of the cylindrical surfaces, determining which partial- each multiperspective panorama; and 

lar location of the group has a computed fitness metric using any traditional stereo matching algorithm requiring 

that indicates the prescribed characteristic of said pro- 25 a horizontal epipolar geometry to designate the coor- 

jected pixels matches more closely than the rest diQates of me ioQ of the scene d icted b each 

(hereinafter referred to as the winning pixel location), . « , - ... 

v , _ • if c.x! corresponding pixel location of the multiperspective 

wherein a group of corresponding pixel locations of the r or r r 

cylindrical surfaces are those which depict the same panoramas. . 

portion of the scene; and 30 55 * ^ computer-readable memory of claim 44, wherem 

for each winning pixel location, identifying its panoramic the P ro 8 ram module for constructing at least two multiper- 

coordinates and designating these coordinates to be the spective panoramas comprises a sub-module for construct- 

position of the portion of the scene depicted by the ing each multiperspective panorama using columnar images 

pixels projected from the multiperspective panoramas obtained from single perspective images whose viewpoints 

to that location. 35 are all at the same radius from a center of rotation, and 

50. The computer-readable memory of claim 49, wherein wherein said radius is less than approximately 0.7 of the 
the sub-module for determining the winning pixel location distance from the center of rotation and the nearest point in 
comprises a sub-module for employing a correlation-based the mnt depicted in any of the columnar images. 

stereo depth technique to identify which particular location 56 ^ ^wte-vwtobk memory of claim 55, wherein 

of each group of corresponding pixel locations of the 40 ^ m module fof computing a 3D reconstruction of 

cylindrical surfaces has a computed fitness metric that ^ ^ rises sub _ modules for: 

indicates the prescribed characteristic of said projected r 

pixels matches more closely than the rest. compensating for any global vertical scale difference in 

51. The computer-readable memory of claim 49, wherein eacn multiperspective panorama; and 

the sub-module for determining the winning pixel location 45 using any traditional stereo matching algorithm requiring 

comprises a sub-module for employing a global optimiza- a horizontal epipolar geometry to designate the coor- 

tion stereo depth technique to identify which particular dinates of the portion of the scene depicted by each 

location of each group of corresponding pixel locations of corresponding pixel location of the multiperspective 

the cylindrical surfaces has a computed fitness metric that panoramas. 

indicates the prescribed characteristic of said projected 50 57. The computer-readable memory of claim 56, wherein 

pixels matches more closely than the rest. thefe m more thaQ two mu itiperspective panoramas used to 

52. Hie computer-readable memory of claim 49, wherein me ^ 3D reconstruction of ^ mm and wher ein the 
the program module for computing a 3D reconstruction of traditional ster eo matchmg aJgorimm is a multi-image stereo 
the scene further comprises a sub-module for spatially . . 

aggregating the computed fitness metric at each pixel loca- 55 ™r I , . 1 e 1 • a a u 

tion on each cylindrical surface, prior to performing the u 58 ' ™ e computer-readable memory of claim 44, wherem 

process action of determining the winning pixel location for me P ro S ram module for ^ructmg at least two multiper- 

each respective group of corresponding pixel locations of SPecUve panoramas comprises a sub-module for construct- 

the cylindrical surfaces. ™& each multiperspective panorama using columnar images 

53. The computer-readable memory of claim 44, wherein 6 0 obtained from single perspective images whose viewpoints 
the program module for constructing at least two multiper- are a11 at tne fixed radms from a center of rotation and 
specuve panoramas comprises a sub-module for construct- which are all laterally centered about a line forming an 
ing a pair of symmetric multiperspective panoramas, off-axis angle with respect to a swing line, said swing fine 
wherein two multiperspective panoramas are symmetric being defined by the center of rotation and the viewpoint 
whenever, 65 associated with each single-perspective images from which 

the columnar images used to construct each panorama are the respective columnar images were obtained, and wherein 

all obtained from single perspective images whose the off-axis angle does not exceed approximately 15 degrees. 
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59. The computer-readable memory of claim 58, wherein 
the program module for computing a 3D reconstruction of 
the scene comprises sub-modules for: 

compensating for any global vertical scale difference in 
each multiperspective panorama; and 

using any traditional stereo matching algorithm requiring 
a horizontal epipolar geometry to designate the coor- 
dinates of the portion of the scene depicted by each 
corresponding pixel location of the multiperspective 
panoramas. 



9,596 Bl 

30 

60. The computer-readable memory of claim 59, wherein 
there are more than two multiperspective panoramas used to 
compute the 3D reconstruction of the scene, and wherein the 
traditional stereo matching algorithm is a multi -image stereo 

5 matching process. 

61. The computer-readable memory of claim 44, further 
comprising a program module for creating a depth map of 
the scene from the computed 3D reconstruction. 

* * * * * 
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